Designing a Benchmark for the Assessment of XML Schema Matching Tools

Over the years, many XML schema matching systems have been developed. A benchmark for assessing the capabilities of schema matching systems and providing uniform conditions and the same testbed for all schema matching prototypes, has become indispensable as the matching systems grow in complexity. However, developing a benchmark for the schema matching problem is very challenging, given the wide range of techniques that can be applied to assist in schema matching. In this paper, we present the foundations and desiderata of a benchmark for XML schema matching. Moreover, we have extended the notion of quality of an integrated schema by proposing new scoring functions. Finally, we have designed and implemented XBenchMatch, an application which takes as input: an ideal schema and the result of a matching from a schema matching prototype (i.e. a set of mappings and/or an integrated schema) and generates as output: statistics on the quality of this input. Our proposal is aimed to provide two kinds of evaluations: (i) quality matching evaluation, which is based on the use of the quality measures and (ii) performance of matching schema. The first criteria is very important in automatic schema matching and the second is crucial in large scale when the schema to be matched are very large. In this paper, we present XBenchMatch, a benchmark for testing and assessing schema matching tools and report the experiments results of some matching tools over a large corpus of schemas using our benchmark.